Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models

摘要

In this paper, we present a deep reinforcement learning (RL) framework foriterative dialog policy optimization in end-to-end task-oriented dialogsystems. Popular approaches in learning dialog policy with RL include letting adialog agent to learn against a user simulator. Building a reliable usersimulator, however, is not trivial, often as difficult as building a gooddialog agent. We address this challenge by jointly optimizing the dialog agentand the user simulator with deep RL by simulating dialogs between the twoagents. We first bootstrap a basic dialog agent and a basic user simulator bylearning directly from dialog corpora with supervised training. We then improvethem further by letting the two agents to conduct task-oriented dialogs anditeratively optimizing their policies with deep RL. Both the dialog agent andthe user simulator are designed with neural network models that can be trainedend-to-end. Our experiment results show that the proposed method leads topromising improvements on task success rate and total task reward comparing tosupervised training and single-agent RL training baseline models.